Using Email Rules for filtering

#1

Is there a way using email integration to track and bucket emails that come in with ‘INFO’ or ‘WARN’ and use specific rules for these events? I see this is possible in Event Rules, but does not appear to be the case in Email Rules. In Email Rules I can only tell it to disregard an email with certain words, not apply specific rules based on these words.

We only have critical alerts going through PagerDuty currently that automatically page the on-call for the responsible team. To better use the metrics available in PagerDuty, we were looking at setting up a service in PagerDuty for every micro-service we have on our stack. We had planned on then being able to track all lower severity alerts and pageable alerts in PagerDuty allowing us to see what is noise and better tweak our alerting.

#2

Hey Gavin! At the moment, we currently do not offer this functionality with the email integration. However, I would not mind at all submitting a feature request for you so our product team is aware of this request. If you like, another way to accomplish your goal is to set up urgencies on your account. Urgencies can be configured to notify you based on your notification rules. When both high and low urgency incidents have been triggered, PagerDuty makes it easy to pinpoint which incidents are high urgency and which are low urgency by color coding them and prioritizing them according to their urgency level. Hope this helps out Gavin! Cheers!

#3

Thank you for the timely response Jay! We do utilize urgencies already for handling issues that don’t require off hours intervention. With this information known I can press the larger Operations team to start working on utilizing the PagerDuty integrations for Splunk and our other monitoring tools, allowing us the more advanced event rules.

Thanks again Jay!
-Gavin

#4

Hey Gavin! No problem at all! We are here to help.

#5

Gavin, There is a way you could achieve what you are looking to do. You can configure multiple services to use the same incoming email address. Then use the email filters to decide whether to create the incident or discard it. In your case you would have a service that has a filter that checks for ‘CRITICAL’ or ‘ERROR’ and if that exists the service creates an incident using high urgency rules. The second service would have a filter that looks for ‘INFO’ or ‘WARN’ and would ignore the emails that come in with the ‘CRITICAL’ and ‘ERROR’ messages. If an email comes in with ‘INFO’ then the first service ignores it and the second service will alert using low urgency. Here is what one might look like.

image

#6

That is another solution that could potentially work, thanks for the idea Kelly. Right now we’re in the process of moving from one main SRT service getting all of the alerts to a one to one service for every application. Creating more services to handle lower priority adds a fair bit of complexity to the integration we were hoping to avoid, unfortunately.

Right now I’m pressing the official PagerDuty integration for Splunk with our Splunk team; if this doesn’t come to fruition I will most likely take your advice and create lower priority/higher priority services like some of our ASM teams currently have implemented.

Thank you Kelly!
Gavin

#7

Another question around this. I recently noticed a section named ‘Incident Priorities’ in the Configuration drop down menu. Can these be used with a simple email integration or do these require the API integration as well? I’m still pushing the admins for implementation of the API solution, but was trying to see just how far I could take the email integration while waiting.
38%20AM

#8

Hi Gavin! Incident Priorities can be used with an email integration on any given account. If you would like to test this out in your account, please let us know so we can enable this feature for you. Also, here is a guide that will shed some light on how this feature works in the UI.

#9

Thanks Jay. I must be missing something here. I modified our Priority levels making the lowest(P5) “INFO” instead. So this should be a warning event, non pageable if I’m reading above correctly. When I created a test incident under priority “INFO” it instantly paged and still had Urgency as High. I have it going to a lower-priority service which is setup to “Use alert severity to determine how responders are notified for each incident” which once again states Info should be a Low Priority incident. What am I missing?
34%20PM

#10

Hey Gavin! Hope you had a great weekend! Would you mind me asking for the incident ID for the test incident you created so I may take a look? Thanks ahead of time!

#11

Incident #69797. Also, the service is setup to “use alert severity to determine how responders are notified for each incident” which if I’m understanding this correctly should be Low urgency but no matter what I do, every alert is always High urgency.

32%20PM

#12

Hey Gavin! Would you mind me asking for your account name you tested this incident on. I am unable to locate on the account on my end.

#13

Hi @GavinClark,

Incident Priorities are more like labels and they don’t determine how Urgency is set.

The Alert Severity to Notification Urgency Mapping uses the CEF (Common Event Fields) from the alert itself, not the labels in Incident Priorities.

I don’t think this (Alert Severity) can be set from an email source, but there are existing mappings for Splunk (which I see you’re using):

Good luck!
@simonfiddaman

#14

Jay, I’m not sure of the exact account name. Bbyops is I believe the account name.

Thank for the information Simon! I’m still working with our team to get the actual Splunk integration going, while I was waiting for that to be implemented I thought I would see what is possible with our current email integration. It seems a bit limited based on what we’d like to do going forward though.

Again, thanks for everyones help.

1 Like
#15

Hi Gavin,

Thanks for providing that!

As Simon mentioned, at the moment incident priorities work more as labels instead of affecting how responders are notified (high-urgency vs. low-urgency).

The best solution in this case, as Kelly described, would be to create separate services, one for high-urgency and one for low-urgency. Based off of this, you can filter the emails for each service looking at your fields “INFO” and “WARN”, etc.

Let us know if you have any additional questions!

#16

Hi @GavinClark

I wanted to resurface this thread to make you aware of another potential solution for your use case. We recently released our Event Rules feature, which allows you to configure rules at the global level determine which actions to perform on an alert as well as which service to route the alert to.

In your case, it sounds like you wanted to determine the severity level of an email event based off of text contained within the email. This is now possible with Event Rules - you can set a severity for an email-based event in order to differentiate which emails should notify your users, and which should be suppressed or flagged as low-urgency. You can read more about how this works here.

I hope this helps - let me know if you have any questions around this.